Image sensing system for a vehicle

ABSTRACT

An image sensing system for a vehicle includes an imaging sensor and a logic and control circuit. The imaging sensor comprises a two-dimensional array of light sensing photosensor elements formed on a semiconductor substrate, and has a field of view exterior of the vehicle. The logic and control circuit comprises an image processor for processing image data derived from the imaging sensor. The image sensing system may generate an indication of the presence of an object within the field of view of the imaging sensor. Preferably, video images may be captured by said imaging sensor and may be displayed by a display device for viewing by the driver when operating the vehicle. The logic and control circuit may generate at least one control output, and the at least control output may control an enhancement of the video images.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 10/643,602, filed Aug. 19, 2003 (Attorney Docket DON01 P-1087),which is a continuation of application Ser. No. 09/776,625, filed onFeb. 5, 2001, now U.S. Pat. No. 6,611,202, which is a continuation ofapplication Ser. No. 09/313,139, filed on May 17, 1999, now U.S. Pat.No. 6,222,447, which is a continuation of application Ser. No.08/935,336, filed on Sep. 22, 1997, now U.S. Pat. No. 5,949,331, whichis a continuation of application Ser. No. 08/445,527, filed on May 22,1995, now U.S. Pat. No. 5,670,935, which is a continuation-in-part ofpatent application Ser. No. 08/023,918, filed Feb. 26, 1993, by KennethSchofield and Mark Larson, now U.S. Pat. No. 5,550,677.

BACKGROUND OF THE INVENTION

This invention relates generally to vision systems for vehicles and,more particularly, to rearview vision systems which provide the vehicleoperator with scenic information in the direction rearward of thevehicle. More particularly, the invention relates to a rearview visionsystem utilizing image capture devices, such as CMOS imaging arrays andthe like.

A long-felt need in the art of vehicle rearview vision systems has beento eliminate exterior rearview mirrors by utilizing image capturedevices, such as cameras, in combination with dashboard displays. Thiswould be beneficial because it would reduce wind drag on the vehicle,wind noise and vehicle weight. Furthermore, rearview mirrors protrude asubstantial distance from the side of the vehicle, which makesmaneuvering in tight spaces more difficult. Image capture devices arecapable of positioning in a greater variety of locations on the vehicle,providing more flexibility of vehicle styling. It is further expectedthat camera systems would greatly reduce the blind spots to the sidesand rear of the vehicle common with vehicles equipped with conventionalrearview mirror systems. The driver cannot perceive vehicles, objects,or other road users in such blind spots without turning his or her body,which interferes with forward-looking visual activities.

Camera-based rearview vision systems for vehicles have not obtainedcommercial acceptance. One difficulty with proposed systems has beenthat they present a large amount of visual information in a manner whichis difficult to comprehend. This difficulty arises from many factors. Inorder to significantly reduce blind spots, multiple image capturedevices are typically positioned at various locations on the vehicle.The image of an object behind the equipped vehicle is usually capturedby more than one image capture device at a time and displayed inmultiple images. This may confuse the driver as to whether more than oneobject is present. When multiple image capture devices are positioned atdifferent longitudinal locations on the vehicle, objects behind thevehicle are at different distances from the image capture devices. Thisresults in different image sizes for the same object. This effect isespecially noticeable for laterally extending images, such as a bridge,highway crosswalk markings, the earth's horizon, and the like. Suchimages are at different vertical angles with respect to the imagecapture devices. This results in different vertical positions on thedisplay causing the elongated image to appear disjointed.

A camera system provides a monocular view of the scene, compared to thebinocular, or stereoscopic, view obtained when the scene is viewedthrough a rearview mirror. This makes the ability to judge distances ina camera system a problem. This effect is most noticeable at distancesclose to the vehicle where stereoscopic imaging is relied uponextensively by the driver in judging relative locations of objects.Therefore, known camera systems fail to provide to the driver importantinformation where that information is most needed—at small separationdistances from surrounding objects.

Another difficulty with camera systems is that, in order to provide asufficient amount of information, the camera system typically presentsthe driver with a greatly increased field of view. This improvesperformance by further reducing blind spots at the side and rear of thevehicle. However, an increased field of view is often obtained byutilizing a wide-angle lens which introduces distortion of the scene andfurther impairs the ability of the driver to judge distances of objectsdisplayed. The problem with such distortion of the scene is that thedriver must concentrate more on the display and take a longer time tointerpret and extract the necessary information. This further distractsthe driver from the primary visual task of maintaining awareness ofvehicles and other objects in the vicinity of the driven vehicle.

SUMMARY OF THE INVENTION

The present invention is directed towards enhancing the interpretationof visual information in a rearview vision system by presentinginformation in a manner which does not require significant concentrationof the driver or present distractions to the driver. This isaccomplished according to the invention in a rearview vision systemhaving at least two image capture devices positioned on the vehicle anddirected rearwardly with respect to the direction of travel of thevehicle. A display is provided for images captured by the image capturedevices. The display combines the captured images into an image thatwould be achieved by a single rearward-looking camera having a viewunobstructed by the vehicle. In order to obtain all of the necessaryinformation of activity, not only behind but also along side of thevehicle, the virtual camera should be positioned forward of the driver.The image synthesized from the multiple image capture devices may have adead space which corresponds with the area occupied by the vehicle. Thisdead space is useable by the driver's sense of perspective in judgingthe location of vehicles behind and along side of the equipped vehicle.

The present invention provides techniques for synthesizing imagescaptured by individual, spatially separated, image capture devices intosuch ideal image, displayed on the display device. This may beaccomplished according to an aspect of the invention by providing atleast three image capture devices. At least two of the image capturedevices are side image capture devices mounted on opposite sides of thevehicle. At least one of the image capture devices is a center imagecapture device mounted laterally between the side image capture devices.A display system displays an image synthesized from outputs of the imagecapture devices. The displayed image includes an image portion from eachof the image capture devices. The image portion from the center imagecapture device is vertically compressed.

It has been discovered that such vertical compression substantiallyeliminates distortion resulting from the spatial separation between thecameras and can be readily accomplished. In an illustrated embodiment,the image compression is carried out by removing selective ones of thescan lines making up the image portion. A greater number of lines areremoved further away from the vertical center of the image.

The compression of the central image portion produces a dead space inthe displayed image which may be made to correspond with the area thatwould be occupied by the vehicle in the view from the single virtualcamera. Preferably, perspective lines are included at lateral edges ofthe dead space which are aligned with the direction of travel of thevehicle and, therefore, appear in parallel with lane markings. Thisprovides visual clues to the driver's sense of perspective in order toassist in judging distances of objects around the vehicle.

According to another aspect of the invention, image enhancement meansare provided for enhancing the displayed image. Such means may be in theform of graphic overlays superimposed on the displayed image. Suchgraphic overlap may include indicia of the anticipated path of travel ofthe vehicle which is useful in assisting the driver in guiding thevehicle in reverse directions. Such graphic overlay may include adistance grid indicating distances behind the vehicle of objectsjuxtaposed with the grid.

These and other objects, advantages, and features of this invention willbecome apparent by review of the following specification in conjunctionwith the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top plan view of a vehicle having a rearview vision systemaccording to the invention;

FIG. 2 is a side elevation of the vehicle in FIG. 1;

FIG. 3 is a front elevation of a display according to the invention;

FIG. 4 is the same view as FIG. 1 illustrating an alternative embodimentof the invention;

FIG. 5 is a block diagram of an electronic system according to theinvention;

FIG. 6 is the same view as FIG. 3 illustrating an alternate mode ofoperation of the system;

FIG. 7 is the same view as FIG. 2 illustrating an alternative embodimentof the invention;

FIG. 8 is the same view as FIG. 3 illustrating an alternative embodimentof the invention;

FIG. 9 is the same view as FIGS. 1 and 4 illustrating an alternativeembodiment of the invention;

FIG. 10 is the same view as FIGS. 3 and 8 illustrating an alternativeembodiment of the invention; and

FIG. 11 is a chart illustrating the horizontal row of pixels (n1, n2) onwhich an object will be imaged from two longitudinally separated imagecapture devices as that object is spaced at different longitudinaldistances from the image capture devices.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now specifically to the drawings, and the illustrativeembodiments depicted therein, a vehicle 10, which may be an automobile,a light truck, a sport utility vehicle, a van, a bus, a large truck, orthe like includes a rearview vision system, generally illustrated at 12,for providing a driver of the vehicle with a view rearwardly of thevehicle with respect to the direction of travel D of the vehicle (FIG.1). Vision system 12 includes at least two side image capture devices 14positioned, respectively, on opposite sides of vehicle 10 and a centerimage capture device 16 positioned on the lateral centerline of thevehicle. All of the image capture devices are directed generallyrearwardly of the vehicle. Rearview vision system 12 additionallyincludes an image processor 18 for receiving data signals from imagecapture devices 14, 16 and synthesizing, from the data signals, acomposite image 42 which is displayed on a display 20.

As will be set forth in more detail below, the images captured by imagecapture devices 14, 16 are juxtaposed on display 20 by image processor18 in a manner which approximates the view from a single virtual imagecapture device positioned forwardly of the vehicle at a location C andfacing rearwardly of the vehicle, with the vehicle being transparent tothe view of the virtual image capture device. Vision system 12 providesa substantially seamless panoramic view rearwardly of the vehiclewithout duplicate or redundant images of objects. Furthermore,elongated, laterally-extending, objects, such as the earth's horizon,appear uniform and straight across the entire displayed image. Thedisplayed image provides a sense of perspective, which enhances theability of the driver to judge location and speed of adjacent trailingvehicles.

Each of side image capture devices 14 has a field of view 22 and isaimed rearwardly with respect to the vehicle about an axis 24 which isat an angle, with respect to the vehicle, that is half of the horizontalfield of view of the image capture device. In this manner, each of theimage capture devices 14 covers an area bounded by the side of thevehicle and extending outwardly at an angle defined by the horizontalfield of view of the respective side image capture device. Center imagecapture device 16 has a horizontal field of view 26, which issymmetrical about the longitudinal axis of the vehicle. The field ofview of each side image capture device 14 intersect the field of view ofcenter image capture device 16 at a point P which is located a distanceQ behind vehicle 10.

Rear blind zones 30 are located symmetrically behind vehicle 10extending from the rear of the vehicle to point P. Side blind zones 25located laterally on respective sides of the vehicle extend rearwardlyof the forward field of view 36 of the driver to the field of view 22 ofthe respective side image capture device 14. An object will not becaptured by side image capture devices 14 or center image capturedevices 16 if the object is entirely within one of the blind zones 25,30. In order for an object, such as another vehicle V or other road usertravelling to the side of vehicle 10, to be observed by an operator ofvehicle 10, the object must be either at least partially within theforward field of view 36 of the driver or be captured by image capturedevices 14, 16 and displayed on display 20. FIG. 4 illustrates vehicle10 travelling on a three-lane highway having lanes L1, L2, and L3 withthe vehicle in lane L2. Another vehicle V is shown positioned mostlywithin one of the blind zones 25, but with the rearmost portion of thevehicle V extending into field of view 22 where the vehicle image willbe captured by one of side image capture devices 14. In the illustratedembodiment, vehicle V is a motorcycle travelling in the center of lanesL1 or L3 and represents a worst case for observing a vehicle travellingat least partially within one of the blind zones 25. In order for aportion of vehicle V to be extending either forwardly or rearwardly ofthe respective blind zone 25, where the vehicle V may be observed byeither the forward field of view 36 of the driver or by the rearviewvision system 12, the field of view 22 of side image capture devices 14must be sufficiently wide to capture a portion of vehicle V asillustrated in FIG. 4. Preferably, the horizontal field of view 22 ofside image capture devices 14 is no greater than that required toprovide sufficient coverage which would be in the range of betweenapproximately 55 degrees and approximately 70 degrees. In theillustrated embodiment, the horizontal field of view 22 is 61 degrees.In order for a portion of vehicle V to be within a vertical field ofview 40 of one of side image capture devices 14, the field of viewshould extend to the pavement at a plane M which intersects vehicle V(FIG. 2). Preferably, vertical field of view 40 is between approximately60 degrees and approximately 75 degrees. In the illustrated embodiment,vertical field of view 40 is 66 degrees. A left overlap zone 32 and aright overlap zone 34 extend rearward from respective points P where thehorizontal fields of view of the side image capture devices intersectthe field of view of center image capture device 16. Overlap zones 32,34 define areas within which an object will be captured both by centerimage capture device 16 and one of the side image capture devices 14. Anobject in an overlap zone 32, 34 will appear on display 20 in multipleimage portions in a redundant or duplicative fashion. In order to avoidthe presentation of redundant information to the driver, and therebyavoid confusion and simplify the task of extracting information from themultiple images or combined images on display 20, the object shouldavoid overlapping zones 32, 34. In practice, this may be accomplished toa satisfactory extent by moving points P away from the vehicle andthereby increasing distance Q. It is desirable to increase distance Q toa length that will exclude vehicles travelling at a typical separationdistance behind vehicle 10 from overlapping zones 32, 34. Thisseparation distance is usually a function of the speed at which thevehicles on the highway are travelling. The faster the vehicles aretravelling, the further Q should be moved behind vehicle 10 to keepoverlap zones 32 and 34 outside of the recommended vehicle spacing. If,however, the vehicles are travelling at a slower speed, then thegenerally accepted recommendation for vehicle spacing decreases and itis more likely that a vehicle will be within overlap zone 32, 34.Therefore, the distance Q may be selected to accommodate expectedvehicle spacing for an average driving speed of vehicle 10.

Distance Q is a function of the effective horizontal field of view 26 ofcenter image capture device 16. As field of view 26 decreases, points Pmove further rearward of the vehicle from a distance Q1, to a distanceQ2, as best illustrated in FIG. 4. In order to increase distance Q toeliminate redundant and duplicative information displayed on display 20for most driving conditions of vehicle 10, field of view 26 ispreferably less than 12 degrees. In the illustrated embodiment, field ofview 26 is between 6 and 8 degrees. Alternatively, distance Q may bedynamically adjusted according to some parameter, such as the speed ofvehicle 10. This would allow Q to be greater when the vehicle istravelling at a faster speed, where vehicle separation tends to belarger, and vice versa. Field of view 26 may be adjusted by utilizing aselective presentation of pixels of the captured image in the displayedimage.

Referring to FIG. 3, image display device 20 displays a composite image42 made up of a left image portion 44, a right image portion 46, and acenter image portion 48. Each image portion 44-48 is reversed from theimage as captured by the respective image capture device 14, 16utilizing conventional techniques. These techniques include reading theimage in reverse with the image capture device, writing the image inreverse to display device 20, or reversing the image in image processor18. Left image portion 44 is joined with central image portion 48 at aboundary 50. Central image portion 48 is joined with right image portion46 at a boundary 52. As may best be seen in FIG. 3, the image portionsat boundaries 50 and 52 are continuous whereby composite image 42 is aseamless panoramic view rearwardly of the vehicle. As also is apparentfrom FIG. 3, central image portion 48 is narrower than either left imageportion 44 or right image portion 46. This is a result of reducing thehorizontal field of view 26 of center image capture device 16sufficiently to move points P, and thus overlap zones 32 and 34, asufficient distance behind vehicle 10 to reduce redundant andduplicative images between image portions 44-48. Composite image 42provides a clear image, which avoids confusion and simplifies the taskof extracting information from the multiple image portions 44-48. Asalso may be seen by reference to FIG. 3, display 20 may additionallyinclude indicia such as the readout of a compass 54, vehicle speed 56,turn signals 58, and the like as well as other graphical or videodisplays, such as a navigation display, a map display, and aforward-facing vision system. In this manner, rearview vision system 12may be a compass vision system or an information vision system.

In the embodiment of rearview vision system 12 having a dynamicallyadjusted value of distance Q, the spacing between boundaries 50 and 52will dynamically adjust in sequence with the adjustment of distance Q.Thus, as overlap zones 32, 34 move further away from the vehicle; forexample, in response to an increase in speed of the vehicle, boundarylines 50 and 52 will move closer together and vice versa. In thismanner, composite image 42 is dynamic, having image portions ofdynamically adaptive sizes.

Display 20 is of a size to be as natural as possible to the driver. Thisis a function of the size of the display and the distance between thedisplay and the driver. Preferably, the displayed image simulates animage reflected by a rearview mirror. As such, the size of display 20 isapproximately the combined areas of the three rearview mirrors (oneinterior mirror and two exterior mirrors) conventionally used withvehicles. As best seen by reference to FIG. 2, display 20 is preferablypositioned within the driver's physiological field of view withoutobstructing the view through the windshield. It is known that thedriver's field of view, with the head and eyes fixed forward, extendsfurther in a downward direction than in an upward direction. Display 20could be located above the vertical view through the windshield whereinthe display may be observed at the upward portion of the driver's fieldof view. However, the position for the display illustrated in FIG. 2 ispreferred wherein the display is within the lower portion of thedriver's field of view.

Display 20, in the illustrated embodiment, is a flat panel display, suchas a back-lit liquid crystal display, a plasma display, a field emissiondisplay, or a cathode ray tube. However, the synthesized image could bedisplayed using other display techniques such as to provide a projectedor virtual image. One such virtual display is a heads-up display. Thedisplay may be mounted/attached to the dashboard, fascia or header, orto the windshield at a position conventionally occupied by an interiorrearview mirror.

Although various camera devices may be utilized for image capturedevices 14, 16, an electro-optic, pixelated imaging array, located inthe focal plane of an optical system, is preferred. Such imaging arrayallows the number of pixels to be selected to meet the requirements ofrearview vision system 12. The pixel requirements are related to theimaging aspect ratio of the respective image capture devices, which, inturn, are a function of the ratio of the vertical-to-horizontal field ofview of the devices, as is well known in the art. In the illustratedembodiment, the imaging aspect ratio of side image capture devices 14 is2:1 and the image aspect ratio of central image capture device 16 isvariable down to 0.1:1. Such aspect ratio will produce images which willnot typically match that of commercially available displays. Acommercially available display may be used, however, by leaving ahorizontal band of the display for displaying alpha-numeric data, suchas portions of an instrument cluster, compass display, or the like, asillustrated in FIG. 3.

In the illustrated embodiment, image capture devices 14, 16 are CMOSimaging arrays of the type manufactured by VLSI Vision Ltd. ofEdinburgh, Scotland, which are described in more detail in co-pendingU.S. patent application Ser. No. 08/023,918, filed Feb. 26, 1993, byKenneth Schofield and Mark Larson for an AUTOMATIC REARVIEW MIRRORSYSTEM USING A PHOTOSENSOR ARRAY, now U.S. Pat. No. 5,550,677, thedisclosure of which is hereby incorporated herein by reference. However,other pixelated focal plane image-array devices, which are sensitive tovisible or invisible electromagnetic radiation, could be used. Thedevices could be sensitive to either color or monochromatic visibleradiation or near or far infrared radiation of the type used innight-vision systems. Each image capture device could be a combinationof different types of devices, such as one sensitive to visibleradiation combined with one sensitive to infrared radiation. Examples ofother devices known in the art include charge couple devices and thelike.

Preferably, image capture devices 14 and 16 are all mounted at the samevertical height on vehicle 10, although compromise may be required inorder to accommodate styling features of the vehicle. The horizontal aimof image capture devices 14 and 16 is preferably horizontal. However,the portion of the image displayed is preferably biased toward thedownward portion of the captured image because significantly less usefulinformation is obtained above the horizontal position of the imagecapture devices.

Each image-capturing device 14, 16 is controlled by appropriatesupporting electronics (not shown) located in the vicinity of theimaging array such that, when operating power is supplied, either ananalog or a digital data stream is generated on an output signal linesupplied to image processor 18. The support electronics may be providedpartially on the image chip and partially on associated electronicdevices. For each exposure period, a value indicative of the quantity oflight incident on each pixel of the imaging array during the exposureperiod is sequentially outputted in a predetermined sequence, typicallyrow-by-row. The sequence may conform to video signal standards whichsupport a direct view such that, when a scene is viewed by animage-capturing device, the image presented on a display representsdirectly the scene viewed by the image-capturing devices. However, whenlooking forward and observing a displayed image of a rearward scene, thedriver will interpret the image as if it were a reflection of the sceneas viewed through a mirror. Objects to the left and rearward of thevehicle, as viewed by the rearward-looking camera, are presented on theleft-hand side of the display and vice versa. If this reversal iseffected in image processor 18, it may be by the use of a data storagedevice, or buffer, capable of storing all of the pixel values from oneexposure period. The data is read out of the data storage device in areversed row sequence. Alternatively, the imaging array electronicscould be constructed to provide the above-described reversal at theimage-capturing device or at the display.

Data transmission between image capture devices 14, 16 and imageprocessor 18 and/or between image processor 18 and display 20 may be byelectrically conductive leads or fiber-optic cable. It is possible, forparticular applications, to eliminate image processor 18 and directdrive display 20 from image capture devices 14, 16 at the pixel level.

The data streams from image-capturing devices 14, 16 are combined inimage processor 18 and directly mapped to the pixel array of display 20.This process is repeated preferably at a rate of at least 30 times persecond in order to present an essentially real time video image. Theimage captured by side image capture device 14 on the right side of thevehicle is presented in right image portion 46 and the image from sideimage capture device 14 on the left side of the vehicle is displayed onleft image portion 44. The image from center image capture device 16 isdisplayed on central image portion 48. The three image portions 44-48are presented in horizontal alignment and adjacent to each other.However, the composite image may be positioned at any desired verticalposition in the display 20. It is also possible to display imageportions 44-48 on separate image devices which are adjacent each other.

In vision system 12, side image capture devices 14 are positionedpreferably at a forward longitudinal position on vehicle 10 and centerimage capture device 16 is positioned at a rearward longitudinalposition on the vehicle. As best seen by reference to FIG. 7, thispositioning creates a difference in the vertical angle between each sideimage capture device 14 and center image capture device 16 with respectto a fixed location P1 that is a distance D1 behind the vehicle. Thisdifference in sensing angle will cause each side image capture device 14to image an object located at P1 on a horizontal row of pixels that isdifferent from the horizontal row of pixels that center image capturedevice 16 will image the same object. If the image is below thehorizontal centerline of the image capture device, it will be imaged ona lower row of pixels by center image capture device 16 than the row ofpixels it will be imaged by the side image capture devices 14, asillustrated in FIG. 7. This mismatch between horizontal pixel rows ofthe captured image is furthermore a function of the distance of thecaptured image from the rear of the vehicle. This can be understood byreference to FIG. 11 which presents a chart 90 having a first column 92of pixel lines n1, measured from the array centerline, at which anobject will be imaged by side image capture device 14 and a secondcolumn 94 of pixel lines n2, measured from the array verticalcenterline, at which the same object will be imaged by center imagecapture device 16. The result is that an object, which is captured byboth side and center image capture devices 14, 16, will be verticallydisjointed at the boundary of the displayed image, if the object iscaptured by more than one image capture device. The amount ofdisjointment will be greater closer to the vehicle and less at furtherdistances. If the object is elongated in the horizontal direction, suchas earth's horizon, bridges, or cross-markings on highways, then theobject will appear to be either broken or crooked.

In order to provide uniform display of laterally elongated images, arearview vision system 12′ is provided having a central image portion48′ which is processed differently from the image display portions 44′and 46′ produced by the side image capture devices (FIG. 8). Centralimage portion 48′ is reduced vertically, or compressed, by removingspecified scan lines, or pixel rows, from the image captured by centerimage capture device 16 in a graduated fashion. The difference in thepixel line at which an object will be imaged by each of the side andcenter image capture devices is a function of the distance D of theobject from the rear of the vehicle, with a greater variation occurringat shorter distances and the variation reducing to zero for infinitedistances. Therefore, the compression of central image portion 48′ isnon-linear, with substantially no compression at the vertical center ofthe image and greater compression at greater distances above and belowthe vertical center point of the image. This is accomplished by removingspecific lines from the center display in a graduated fashion with agreater number of lines removed further from the vertical center of theimage. The removed lines may be merely discarded in order to verticallyreduce the image. Alternatively, the data contained in the removed linesmay be utilized to modify the value of adjacent pixels above and belowthe removed line in order to enhance the quality of the compressedimage. Averaging, median filtering, or other such known techniques mayalso be used.

Each of right image portion 46′ and left image portion 44′ includes anupper portion 64 which extends above the compressed upper portion of thecentral image portion 48′. In the illustrated embodiment, upper portions64 are deleted in order to present a uniform upper horizontal boundaryfor display 20′. In the illustrated embodiment, the mismatch between thelower horizontal boundary of central image portion 48′ and each of theleft and right image portions provides a dead space 66 which provides avisual prompt to the user of the approximate location of the rearwardcorners S of vehicle 10. This dead space 66 in the image displayed ondisplay 20′ approximates the footprint occupied by vehicle 10 whenviewed from point C. This is particularly useful because it provides avisual indication to the driver that a vehicle passing vehicle 10, asviewed in either left image portion 44′ or right image portion 46′, isat least partially adjacent vehicle 10 if the image of the approachingvehicle is partially adjacent to dead space 66.

In an alternative embodiment, the vertical compression technique may beapplied to only a lower vertical portion of central image portion 48′.In most driving situations, objects imaged by rearward-facing imagecapture devices above the horizon are at a long distance from thevehicle while those below the horizon get progressively closer to thevehicle in relation to the distance below the horizon in the displayedimage. Therefore, compression of the upper vertical portion of thecentral image portion may be eliminated without significant reduction inperformance.

Compression of the central image portion may also advantageously beprovided horizontally, as well as vertically. Spatial separation ofcenter image capture device 16 from side image capture devices 14 causessimilar distortion, as that described above, in the horizontaldirection. This effect is spherical in nature and would require a morecomplex corrective action, such as compressing the image based upon theremoval of pixels from an approximation to concentric circles centeredon the center of the imaging array, or other techniques which would beapparent to those skilled in the art.

A rearview vision system 12″ includes an image display 20″ having acompressed central image portion 48″ and left and right image portions44″ and 46″, respectively (FIG. 10). A border 50′ between left sideimage 44″ and central image 48″ includes a vertical central borderportion 50 a′, an upper border portion 50 b′, and a lower border portion50 c′. Upper border portion 50 b′ and lower border portion 50 c′ divergelaterally outwardly, vertically away from central portion 50 a′. Aborder 52′ between central image portion 48″ and right image portion 46″includes a central boundary portion 52 a′, an upper boundary portion 52b′, and a lower boundary portion 52 c′. Upper boundary portion 52 b′ andlower boundary portion 52 c′ diverge laterally outwardly vertically awayfrom central portion 52 a′. This creates an upper portion of centralimage portion 48″ and a lower portion of central image portion 48″ whichextend beyond the center portion thereof. This configuration is basedupon the realization that the surface of the road immediately behind thevehicle is captured by central image capture device 16. Likewise, thehorizontal plane above the vehicle, which is symmetrical with the roadsurface, is captured by the center image capture device. This may beseen by referring to point P in FIG. 10, which illustrate the pointswhere the effective radius 68 of the virtual image capture deviceintersects dead zones 30 and by referring to point S in FIG. 10 whichillustrates the comers or the rear of the vehicle (S).

The image displayed on display 20″ includes a dead space 66′ havingdiverging lateral sides 68 a, 68 b. Diverging sides 68 a and 68 b areconfigured in order to extend in the direction of travel of vehicle 10which is parallel to lane markings of a highway on which vehicle 10 istravelling. This further enhances the visual perception of the driver byproviding a visual clue of the location of images appearing on display20″ with respect to the vehicle 10. Side portions 68 a, 68 b, in theillustrated embodiment, are natural extensions of lower boundaryportions 50 c′ and 52 c′ and extend from point S on each respective sideof the vehicle to point R, which represents the intersection of thelower extent of the vertical field of view 40 of each side image capturedevice 14 with the pavement (FIG. 7).

Rearview vision systems 12′ and 12″ utilize a displayed synthesizedimage which takes into account the use of perspective in enhancing thedriver's understanding of what is occurring in the area surrounding thevehicle. The images produced on displays 20′ and 20″ effectively removethe vehicle bodywork and replace the bodywork with a vehicle footprintas would be viewed by virtual camera C. The image displayed on display20″ further includes perspective lines which further enhance the roll ofperspective in the driver's understanding of what is occurring.

In order to further enhance the driver's understanding of what isoccurring in the area surrounding the vehicle, a rearview vision system12″′ includes a display 20″′ having image enhancements (FIG. 6). In theillustrative embodiment, such image enhancements include graphicoverlays 70 a, 70 b which are hash marks intended to illustrate to thedriver the anticipated path of movement of vehicle 10. In theillustrated embodiment, the anticipated vehicle motion is a function ofthe vehicle direction of travel as well as the rate of turn of thevehicle. The forward or rearward direction of vehicle travel isdetermined in response to the operator placing the gear selection device(not shown) in the reverse gear position. The degree of turn of thevehicle may be determined by monitoring the movement of the vehiclesteering system, monitoring the output of an electronic compass, ormonitoring the vehicle differential drive system. In the embodimentillustrated in FIG. 6, the configuration of graphic overlays 70 a, 70 bindicates that the vehicle is in reverse gear and that the wheels areturned in a manner that will cause the vehicle to travel toward thedriver's side of the vehicle. If the wheels were turned in the oppositedirection, graphic overlays 70 a, 70 b would curve clockwise toward theright as viewed in FIG. 6. If the vehicle's wheels were straight,graphic overlays 70 a, 70 b would be substantially straight converginglines. If the vehicle is not in reverse gear position, graphic overlays70 a, 70 b are not presented. Other types of graphic overlays of thedisplayed image are comprehended by the invention.

Horizontal grid markings on the display may be provided to indicatedistances behind the vehicle at particular markings. Such grid wouldallow the driver to judge the relative position of vehicles behind theequipped vehicle. In one embodiment, short horizontal lines aresuperimposed on the displayed image at regular rearward intervals inhorizontal positions which correspond to the boundaries of the lane inwhich the vehicle is travelling. In order to avoid confusion when thevehicle is travelling in a curved path, from a lack of correspondencebetween the graphic overlay and the road, a signal indicative of thevehicle's rate of turn may be taken into account when generating thegraphic overlay. In this manner, the distance indications may be movedlaterally, with reduced horizontal separation, to correspond to thepositions of the curved lane boundaries and vertically on the image tocompensate for the difference between distances along a straight andcurved path.

Another image enhancement is to alter the appearance of an object in aparticular zone surrounding the vehicle in order to provide anindication, such as a warning, to the driver. As an example, a vehiclethat is too close to the equipped vehicle for safe-lane change, may bedisplayed in a particular color, such as red, may flash, or otherwise bedistinguishable from other images on the display. Preferably, the speedof the equipped vehicle 10, which may be obtained from known speedtransducers, may be provided as an input to the rearview vision systemin order to cause such warning to be a function of the vehicle speedwhich, in turn, affects the safe separation distance of vehicles. Theoperation of the turn signal may also be used to activate suchhighlighting of other road users or to modify the scope of the imagedisplayed. In order to determine the distance of objects behind vehicle10, a separate distance-measuring system may be used. Such separatesystem may include radar, ultrasonic sensing, infrared detection, andother known distance-measuring systems. Alternatively, stereoscopicdistance-sensing capabilities of side image capture devices 14 may beutilized to determine the separation distance from trailing objectsutilizing known techniques.

Thus, it is seen that the image displayed on display 20-20′″ may bedifferent under different circumstances. Such different circumstancesmay relate to the vehicle's direction of travel, speed, rate of turn,separation from adjacent objects, and the like.

Various other forms of image processing may be utilized with rearviewvision system 12-12″′. Luminant and clirominant blending may be appliedto the images captured by image capture devices 14, 16 in order toproduce equality of the image data whereby the image portions appear asif they were produced by one image capture device. The dynamic range ofthe image capture devices may be extended in order to provide highquality images under all lighting conditions. Furthermore, individualpixel groups may be controlled in order to selectively compensate forbright or dark spots. For example, anti-blooming techniques may beapplied for bright spots. Multiple exposure techniques may be applied tohighlight dark areas. Image morphing and warping compensation techniquesmay additionally be applied. Resolution of the image capture devices anddisplay may be selected in order to provide sufficient image quality forthe particular application.

A heater may be applied to each image capture device in order to removedew and first that may collect on the optics of the device. Although, inthe illustrative embodiment, the optical centerline of the cameracoincides with the field of view, particular applications may result inthe centerline of the camera pointing in a direction other than thecenterline of the field of view. Although, in the illustrativeembodiment, the image capture devices are fixed, it may be desirable toprovide selective adjustability to the image capture devices or opticalpaths in particular applications. This is particularly desirable whenthe system is used on articulated vehicles where automated andcoordinated camera aim may be utilized to maintain completeness of thesynthesized image.

When operating the vehicle in the reverse direction, it may be desirableto provide additional data concerning the area surrounding the immediaterear of the vehicle. This may be accomplished by utilizingnon-symmetrical optics for the center image capture device in order toprovide a wide angle view at a lower portion of the field of view.Alternatively, a wide angle optical system could be utilized with theelectronic system selectively correcting distortion of the capturedimage. Such system would provide a distortion-free image while obtainingmore data, particularly in the area surrounding the back of the vehicle.

The invention additionally comprehends the use of more than three imagecapture devices. In addition to side image capture devices positioned atthe front sides of the vehicle and a center image capture devicepositioned at the center rear of the vehicle, additional image capturedevices may be useful at the rear corners of the vehicle in order tofurther eliminate blind spots. It may additionally be desirable toprovide an additional center image capture device at a higher elevationin order to obtain data immediately behind the vehicle and thereby fillin the road surface detail immediately behind the vehicle. Suchadditional detail is particularly useful when operating the vehicle inthe reverse direction. Of course, each of the image capture devicescould be a combination of two or more image capture devices.

Although the present invention is illustrated as used in a rearviewvision system, it may find utility in other applications. For example,the invention may be useful for providing security surveillance in anarea where a building or other object obstructs the view of the areaunder surveillance. Additionally, the invention may find application innight-vision systems and the like. For example, the invention may beapplied to forward-facing night-vision systems, or other visionenhancement systems such as may be used in adverse weather oratmospheric conditions such as fog, applied to provide an enhanceddisplay of a synthesized image, which approximates a forward-facing viewfrom a single virtual camera located rearwardly of the driver, takingadvantage of the perspective features of the image.

Thus, it is seen that the present invention enhances the relationshipbetween the driver's primary view and the image presented on therearview vision system. This is accomplished in a manner which providesease of interpretation while avoiding confusion so that the driver doesnot have to concentrate or look closely at the image. In this manner,information presented on the display is naturally assimilated. This isaccomplished while reducing blind spots so that other vehicles orobjects of interest to the driver will likely be displayed to thedriver. Additionally, the use of perspective allows distances to be moreaccurately determined.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

1. An image sensing system for a vehicle, said image sensing systemcomprising: an imaging sensor; said imaging sensor comprising atwo-dimensional array of light sensing photosensor elements; whereinsaid array of light sensing photosensor elements and at least a portionof said logic and control circuit are commonly formed on a semiconductorsubstrate; said imaging sensor having a field of view exterior of thevehicle; wherein said image sensing system generates an indication ofthe presence of an object within the field of view of said imagingsensor; wherein said indication comprises at least one of a visualwarning and an audible warning; a logic and control circuit; and saidlogic and control circuit comprising an image processor for processingimage data derived from said imaging sensor.
 2. The image sensing systemof claim 1, wherein said array of light sensing photosensor elements isformed on said semiconductor substrate as a CMOS device.
 3. The imagesensing system of claim 1, wherein said array of light sensingphotosensor elements and at least a portion of said logic and controlcircuit are commonly formed on a semiconductor substrate as anintegrated circuit.
 4. The image sensing system of claim 1, wherein saidarray of light sensing photosensor elements and at least a portion ofsaid logic and control circuit are commonly formed on a semiconductorsubstrate as a CMOS device.
 5. The image sensing system of claim 1,wherein said logic and control circuit generates at least one controloutput, said control output being indicative of another vehicle rearwardof and approaching the vehicle equipped with said imaging sensor.
 6. Theimage sensing system of claim 5, wherein said at least one controloutput is indicative that a lane change may not be safely undertaken bythe vehicle equipped with said imaging sensor.
 7. The image sensingsystem of claim 6, wherein said at least one control output is generatedbased on at least one of distance and speed of a trailing vehicle. 8.The image sensing system of claim 5, wherein said at least one controloutput controls an alarm audible to the driver when operating thevehicle.
 9. The image sensing system of claim 5, wherein said at leastone control output controls a display device visible to the driver whenoperating the vehicle.
 10. The image sensing system of claim 5, whereinvideo images captured by said imaging sensor are displayed by a displaydevice for viewing by the driver when operating the vehicle.
 11. Theimage sensing system of claim 10, wherein the driver of the vehicle hasa field of view through the windshield of the vehicle and wherein saiddisplay device may be observed by the driver at the upward portion ofsaid driver's field of view.
 12. The image sensing system of claim 10,wherein said display device is disposed at or proximate an interiormirror assembly location of the vehicle.
 13. The image sensing system ofclaim 10, wherein said display device comprises a display deviceselected from the group comprising (i) a back lit liquid crystal displaydevice, (ii) a plasma display device, and (iii) a field emission displaydevice.
 14. The image sensing system of claim 10, wherein said at leastone control output controls an enhancement of said video images.
 15. Theimage sensing system of claim 14, wherein said enhancement of said videoimages comprises at least one of (i) a graphic overlay, (ii) displayingan imaged object in a particular color, and (iii) flashing an imagedobject.
 16. The image sensing system of claim 15, wherein saidenhancement is a function of the speed of the vehicle equipped with saidimaging sensor.
 17. The image sensing system of claim 14, wherein saidenhancement of said video images displayed by said display device is afunction of at least one of (i) the travel direction of the vehicleequipped with said imaging sensor, (ii) the speed of the vehicleequipped with said imaging sensor, (iii) the rate of turn of the vehicleequipped with said imaging sensor, and (iv) the separation distance ofthe vehicle equipped with said imaging sensor to adjacent objects. 18.The image sensing system of claim 1, wherein video images captured bysaid imaging sensor are displayed by a display device for viewing by thedriver when operating the vehicle in a forward direction of travel. 19.The image sensing system of claim 1, wherein said logic and controlcircuit comprises at least one of (i) an analog-to-digital converter,(ii) a logic circuit, (iii) a clock, (iv) a random access memory, and(v) a digital-to-analog converter.
 20. The image sensing system of claim1, wherein said field of view comprises a blind spot to the rear of thevehicle.
 21. The image sensing system of claim 1, wherein the vehicleequipped with said imaging sensor comprises a distance-measuring system.22. The image sensing system of claim 21, wherein saiddistance-measuring system comprises at least one of (a) a radar sensor,(ii) an ultrasonic sensor, (ii) an infrared sensor and (ii) astereoscopic sensor.
 23. The image sensing system of claim 1, whereinsaid imaging sensor is disposed at an interior portion of the vehicleand has a field of view exterior of the vehicle through a window of thevehicle.
 24. The image sensing system of claim 1, wherein said imagingsensor comprises a night vision imaging sensor.
 25. The image sensingsystem of claim 24, wherein said night vision imaging sensor comprises amonochrome imaging sensor.
 26. The image sensing system of claim 25,wherein said monochrome night vision imaging sensor is disposed at aninterior portion of the vehicle and has a field of view exterior of thevehicle through a window of the vehicle.
 27. An image sensing system fora vehicle, said image sensing system comprising: an imaging sensor; saidimaging sensor comprising a two-dimensional array of light sensingphotosensor elements; wherein said array of light sensing photosensorelements is formed on a semiconductor substrate as a CMOS device; saidimaging sensor having a field of view exterior of the vehicle; a logicand control circuit; said logic and control circuit comprising an imageprocessor for processing image data derived from said imaging sensor; adisplay device visible to the driver when operating the vehicle; whereinvideo images captured by said imaging sensor are displayed by saiddisplay device for viewing by the driver when operating the vehicle;wherein said logic and control circuit generates at least one controloutput, and wherein said at least one control output controls anenhancement of said video images; and wherein said enhancement enhancesthe driver's awareness of the presence of an object within the field ofview of said imaging sensor.
 28. The image sensing system of claim 27,wherein said array of light sensing photosensor elements and at least aportion of said logic and control circuit are commonly formed on asemiconductor substrate as at least one of (i) an integrated circuit and(ii) a CMOS device.
 29. The image sensing system of claim 27, whereinsaid control output is a function of a determination by said logic andcontrol circuit of another vehicle rearward of and approaching thevehicle equipped with said imaging sensor.
 30. The image sensing systemof claim 29, wherein said at least one control output is indicative thata lane change may not be safely undertaken by the vehicle equipped withsaid imaging sensor.
 31. The image sensing system of claim 30, whereinsaid indication is generated based on at least one of distance and speedof a trailing vehicle.
 32. The image sensing system of claim 29, whereinsaid at least one control output controls an alarm audible to the driverwhen operating the vehicle.
 33. The image sensing system of claim 27,wherein said logic and control circuit comprises at least one of (i) ananalog-to-digital converter, (ii) a logic circuit, (iii) a clock, (iv) arandom access memory, and (v) a digital-to-analog converter.
 34. Theimage sensing system of claim 27, wherein said video images are visibleto the driver when operating the vehicle in a forward direction oftravel.
 35. The image sensing system of claim 27, wherein said field ofview comprises a blind spot to the rear of the vehicle.
 36. The imagesensing system of claim 27, wherein said display device is disposed ator proximate an interior mirror assembly location of the vehicle. 37.The image sensing system of claim 27, wherein said display devicecomprises a display device selected from the group comprising (i) a backlit liquid crystal display device, (ii) a plasma display device, and(iii) a field emission display device.
 38. The image sensing system ofclaim 27, wherein said enhancement of said video images comprises atleast one of (i) a graphic overlay, (ii) displaying an imaged object ina particular color, and (iii) flashing an imaged object.
 39. The imagesensing system of claim 27, wherein said enhancement is a function ofthe speed of the vehicle equipped with said imaging sensor.
 40. Theimage sensing system of claim 27, wherein the vehicle equipped with saidimaging sensor comprises a distance-measuring system.
 41. The imagesensing system of claim 40, wherein said distance-measuring systemcomprises at least one of (a) a radar sensor, (ii) an ultrasonic sensor,(ii) an infrared sensor and (ii) a stereoscopic sensor.
 42. The imagesensing system of claim 27, wherein said enhancement of said videoimages displayed by said display device is a function of at least one of(i) the travel direction of the vehicle equipped with said imagingsensor, (ii) the speed of the vehicle equipped with said imaging sensor,(iii) the rate of turn of the vehicle equipped with said imaging sensor,and (iv) the separation distance of the vehicle equipped with saidimaging sensor to adjacent objects.
 43. The image sensing system ofclaim 27, wherein said imaging sensor is disposed at an interior portionof the vehicle and has a field of view exterior of the vehicle through awindow of the vehicle.
 44. The image sensing system of claim 27, whereinsaid imaging sensor comprises a night vision imaging sensor.
 45. Theimage sensing system of claim 44, wherein said night vision imagingsensor comprises a monochrome imaging sensor.
 46. The image sensingsystem of claim 45, wherein said monochrome night vision imaging sensorimaging sensor is disposed at an interior portion of the vehicle and hasa field of view exterior of the vehicle through a window of the vehicle.47. An image sensing system for a vehicle, said image sensing systemcomprising: an imaging sensor; said imaging sensor comprising atwo-dimensional array of light sensing photosensor elements; whereinsaid array of light sensing photosensor elements is formed on asemiconductor substrate; said imaging sensor having a field of viewexterior of the vehicle; a logic and control circuit; said logic andcontrol circuit comprising an image processor for processing image dataderived from said imaging sensor; a display device visible to the driverwhen operating the vehicle; wherein video images captured. by saidimaging sensor are displayed by said display device for viewing by thedriver when operating the vehicle; wherein said logic and controlcircuit generates at least one control output, and wherein said at leastcontrol output controls an enhancement of said video images; whereinsaid enhancement enhances the driver's awareness of the presence of anobject within the field of view of said imaging sensor; and wherein atleast one of (i) said field of view comprises a blind spot to the rearof the vehicle and (ii) said control output is a function of adetermination by said logic and control circuit of another vehiclerearward of and approaching the vehicle equipped with said imagingsensor.
 48. The image sensing system of claim 47, wherein said array oflight sensing photosensor elements is formed on said semiconductorsubstrate as a CMOS device.
 49. The image sensing system of claim 47,wherein said array of light sensing photosensor elements and at least aportion of said logic and control circuit are commonly formed on asemiconductor substrate.
 50. The image sensing system of claim 47,wherein said array of light sensing photosensor elements and at least aportion of said logic and control circuit are commonly formed on asemiconductor substrate as at least one of (i) an integrated circuit and(ii) a CMOS device.
 51. The image sensing system of claim 47, whereinsaid control output is a function of a determination by said logic andcontrol circuit of another vehicle rearward of and approaching thevehicle equipped with said imaging sensor and wherein said at least onecontrol output is indicative that a lane change may not be safelyundertaken by the vehicle equipped with said imaging sensor.
 52. Theimage sensing system of claim 51, wherein said indication is generatedbased on at least one of distance and speed of a trailing vehicle. 53.The image sensing system of claim 47, wherein said logic and controlcircuit comprises at least one of (i) an analog-to-digital converter,(ii) a logic circuit, (iii) a clock, (iv) a random access memory, and(v) a digital-to-analog converter.
 54. The image sensing system of claim47, wherein said at least one control output controls an alarm audibleto the driver when operating the vehicle.
 55. The image sensing systemof claim 47, wherein said display device comprises a display deviceselected from the group comprising (i) a back lit liquid crystal displaydevice, (ii) a plasma display device, and (iii) a field emission displaydevice.
 56. The image sensing system of claim 47, wherein saidenhancement of said video images comprises at least one of (i) a graphicoverlay, (ii) displaying an imaged object in a particular color, and(iii) flashing an imaged object.
 57. The image sensing system of claim47, wherein said enhancement is a function of the speed of the vehicleequipped with said imaging sensor.
 58. The image sensing system of claim47, wherein the vehicle equipped with said imaging sensor comprises adistance-measuring system.
 59. The image sensing system of claim 58,wherein said distance-measuring system comprises at least one of (a) aradar sensor, (ii) an ultrasonic sensor, (ii) an infrared sensor and(ii) a stereoscopic sensor.
 60. The image sensing system of claim 47,wherein said enhancement of said video images displayed by said displaydevice is a function of at least one of (i) the travel direction of thevehicle equipped with said imaging sensor, (ii) the speed of the vehicleequipped with said imaging sensor, (iii) the rate of turn of the vehicleequipped with said imaging sensor, and (iv) the separation distance ofthe vehicle equipped with said imaging sensor to adjacent objects. 61.The image sensing system of claim 47, wherein said imaging sensorcomprises a color imaging sensor.